: | Fo 


, rv \ ? 7 * . rai 
DCCOBENT RESUME | -* | a 
aa . as : : . 7 7 
ED 178 567 y ’ 4 * 7 TH 009°711 
AUTHOR ~~: McDermott, Paul A.; Watkins, Marley W. 
’ TITLE - Program STANDARD (Statistic of Conjcint Nuitipie 
0 Observer Agreement with a Standard) . | 
PUB DATE. i 2? r. 
NOTE ages ae _ : ; 
EDRS PRICE AFO1/PCQ1. Plus eockwse. 
DESCRIPTORS | Classification; *Computey Programs; *Gocdness of Fit; 
tf: Nonparametric Statistics; Observation; Rating Seales: 
“Reliability;( Statistical Analysis; *Tests of | 
cars s% Significance : 
IDENTIFIERS | interrateg Reliability: * STAN DARD Computer Program 
ABSTRACT | a 


A comprker: progran aued Progtam STANDARD is ° 
presented ana demonstrated. This’ Frogram calculates the statistical 

ignificance of the overall agreement of the categorical assignments. 
nN e program is based on Light's statistic, G, for describing the 
conjoint agreement of(many observers with corfect or standard set’ of 
classifications om nominal scalés. The present documentation ‘includes 
a description of the program!s Furfoses, input, output, language and 
processing, jimitations and computational formylae. Also provided are 
instructions for preparing the jek deck, a listing cf the FORTRAN 
source deck, order of!the éard deck for computer processing, ‘and a 
cpg ete set of sample. input and. cutput data. (Author/crs) 


» 


J ; 
Ope ok ok a a ae ic SOI IO Ke SE A OR I a OK Re OK OE PO ACK RE FR RRO BO RK AK ae 


* Reproductions supplied by EDRS are the best tnat can be made * 


o* ; grom the criginal document. % 
{FEROS BEEBE DD IEEE III EIEIO AI AAAI IO 


tA 


a ; 3 


6 
Nea! 


‘ # ; 2 F 
~ * : . 
* : . U.S. DEPARTMENT OF HEALTH, 
5 . EDUCATION & WELFARE 
é . RATIONAL INSTITUTE OF 
EOUCATION 


é ‘ THIS DOCUMENT HAS BEEN REPRO. 
* OuUCED EXACTLY AS RECEIVED FROM 
; j . 7 THE PERSON OR ORGANIZATION ORIGIN 
’ ATING HY POINTS OF. VIEW OR OPINIONS 
4 . ? % — : WHATED DO NOT NECESSARILY REPRE- 
‘ : SENT OFFICIAL RATIONAL INSTETUNE OF 

EDUCATION POSITION OR POLICY 

2 


* 


ED178567 


a 


,Program STANDARD 


. 


(Statistic of Conjoint .Multiple Observer Agreement with a Standard) 


“y 


‘*~ > . 
d by. 3 
. Paul A. McDermott ~ and ‘Marley W. Watkins « % - 4 
University of Pennsylvania University of Nebraska-Lincoln _” 
7 ‘ EO .7 \ * 4 5 
. 4 . 7 + 
- ; 2 4 ‘ x f 
“PERMISSION TO REPRODUCE THIS e i 
: ERIAL'HAS BEEN GRANTED BY . 
\ : 7 
: TO THE EDUGATIONAL RESOURCES : 
INFORMATION CENTER (ERIC) 
a ; & 
‘ 
; . rae i ao 
- = . f 
. . . wa oe 
| % 
> : . Py 
22 . > * - tr » 
t 
—* : , 
= e | 
fomne / 
«) a -_ 


@ 
ERIC 


A computer program named Progran STANDARD is presented wid demonstrated. .” 


This program calculates the statistical significance of the overall: agree- 
ment ‘of the categorical assignments of two, or more observers. with a set of. . 
true. or correct categorical assignments. The present documentation includes 


‘a description of the’ program's purposes, input, output, language and process~ 


ing, limitations and computational: formulae. Also provided are instructions 
for preparing the job deck, a listing of ‘the FORTRAN source deck, order; “Of 


the card deck for computer processing, and a complete set of sania input . 
and output data.” ~ ~ 
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COMPUTATIONAL FORMULAE 
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4 - Description and Features 
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Introduction. Savers: measures of response agreement fow observers’ 
or raters' classifications on nominal ‘scales appear to be finding increased 
popularity and a variety of applications among researchers and. ‘practitioners. 
Cohen's (1960) development of the statistic A provided a means. to determine 
the level and significance of the agreement between two observers in their. 


: assignments of objects or subjects to nominal scale categories. ‘Later, — 


Light (1971) refined and extended A for usefin the situation where multiple 
- (4. e., more than two) observers would be making. such aSsignments and further 
provided a method for testing the agreement among such multiple observers, 
upon, each specific: nominal category considered. This multiple observer 
agreement’ statistic, known as m, was further extended by Fleiss (1971) - 


to the case in which the sets of multiple observers would not remain identi- 


cal from one: case to “another. 


* . The several et eee agreement measures share two principal wroper~ 


ties in common: first, they make no assumption that any true, correct or 
standard set of nominal scale classifications might exist against which the 


Be classifications offered by the various observers might be evaluated, and 


second, in cases where versions of K mare applied, there being more than 
two observers, involved,’ the’ assifications of all observers are weighted. 
yequally thus rendering a theasure of the overall conjoint agreement of the 


set of classifications. | “ — 


However, when considering: response agreement among observers in applied | 


',and research settings, it is frequently desireable to test the conjoint 


— 


agreement among observers relative to a standard set of classifjcations. | , 
This is true, for example, whenever it is necessary--to assess the categorizing 
ability of trainee observers in the light of an expert's categorizations or 

when the classification accuracy of a categorical rating device must be sized 

up against the conjoint ratings of indépendent .expert observers. For thtis 
purpose Light (1971) has defined the statistic G to test the significance of 

the conjoint agreement of many observers with a correct or standard set’ of 
classifications on nominal scales. 


- G is based upon a special variation of a maitipie contingency table 
routine which first compares the obtained agreement of each of the observers’ 
categorical choites with the standard categorical choice and thereafter com- 
pares this obtained level of agreement with what would be expected under the 
null hypothesis of fandom assigninent of cases to categories. For purposes 
of testing. statistifal significance, G is distributed ae a according 


to the unit normal deviate. 


4 “ 


The computer program presented and demonstrated in this paper, known as: 
Program STANDARD, tests the statistical significance of the conjoint agree- . se 
ment of two or more observers with a standard set of classifications based 
upon Light’s (1971) computational formulae for the statistic G. 
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. (3) standard category choices for each case; (4) observers’ 
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Input. Each analysis requires four control cards and a data ecard deck’ 
as follows: (1) a tithe card; (2) a problem card to specify the number of. 
cases apie classified, number of categories and number of observers; (3) a 
pair o standard cards indicating the standard category choices for each 
case considered; and (4) a sét of observer cards, one or two per each 
observer,: specifying observers’ eaeeeony choices for each case. 

- Output. The ‘information provided for ack analysis includes: (1) an 
alphanumeric job title; (2) number of cases, categories and observers; - 
categ ‘choices 
for each case; and (5) value of the ‘G statistic (see Equation 1 ff., pp. 5-6) 
and level of statistical significance associated with the unit normal deviate. 

gua: Written in FORTRAN Iv, Program STANDARD is. 
compatible with machine¢ in the IBM 360° series. Variables are in memonic - 
form accordjng to Light (1971) computational formulae. . Input editing and _ 
dutput specifications: are provided for user's syntactical errors. 


Limitations. Currently. Program STANDARD will perait up to 160 cases © : 
to be assigned by 100 or fewer observers to a maximum of 10 nominal - categories. 
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Computational Formulae 
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'“ The -Program STANDARD algorithms are as follows: 
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== i J Instructions for Preparing Job Deck 


are 


Ae, 


The following/ cards ala be prepared: 


ee a Card 1: i ? 7 
Title card--A job title may be punched a columns 1-80 =. 
ad Card ay , 
ue Problem Card--Specifications defined below. All numbers must be 
/ : " -right-adjusted, . ; ; 
_ .Colums °° — Punch — a cs : 
: ; 1-3 Number of cages (maximum = 16 ) 
me 7 f 4&5 Nuriber of categories. (maximum = 10) 
= fo $88 * Number of observers (maximum = 100) s 
Cards’ 3 & 4: 


a 


‘Standard Cardg--A standard: or “correct” category choice must be 
: indicated for each category specified in columns 
Sigs 4 ’ ‘4-5 of the Problem Card.. If the standard category. 
; oe “ choice for the lst case,is category 1, then punch i 
x6 ; am : dn column 1. If*the standard. category. choice for. 
se 5 = . 7 _.  “ the Ist case is category 2, than punch 2 in colum 
oe 1....I£ the standard category choice for the lst 
“a 8 : ; case is category 9,.then punch 9 in colum l. If 
oo! the standard category: choice for the lst case is 
category 10, then punch 0 in column 1. Repeat ~ 
ge - standard category specifications with each colum 
. corresponding to each case specified in columns 1-3 
f-the Problem Card. Two standard cards must be’ 
‘ . provided. If the number of cases is less than 81, 
the second card must be'a blank card. 


Observer Cards--One or two observer cards ‘must be eunvhed for each 
ee —_ . case, If the problem calls for more than 80 cases, 
a : then’ two observer cards must be punched for each 
* case. If the observer category choice for the first 
case is 1, then punch 1 ‘in column 1. Continue as 
outlined in directions for Standard Cards. 


‘ “ . 


For examplé: 5 observers classified 0 cases into 3 categoriesS— 


(Title) EXAMPLE PROBLEM 
(Problem) 01003005 
(Standard 1) 1223332113 | 
* _ (Standard 2) blank card |. . 
. (Observer 1) . 1233222123 | > 
‘ (Observer 2) + 3223311112 ; 
\ ‘ oy es 


>» 


(Observer 5) 1233211133 
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Program Source Listing - - , bear 4, 

oa : a * .: 

7 7 x | 
DIMENSION MATRIX( 1000, 10), SUMROW(-10) »SUMCLM(10) »RATOLM (100,100, > 


~ XSUBS IX (100 » 10), SEVEN1( 10), A010), Eee PHC E LO NNAV LCE E Os 


KKEY(160),TITLE( 20) " 


INTEGER, TMs RATCLMy NRAWs MATRIX? TITLE _? 
DOUBLE PRECISION PART1,ETMyONE, TWO, THREE’, FOUR » VFIVE SEK : 


DATA TM/0/,ROW/0 .00/,COLUYN/O. 007, PART1/0200/,ETM/0.00/ 5 0NE/0. ,00/, 
XTWO/0.00/,THREE/O -00/+ZETA/IO. 00/, FOUR /0.007, FI VE/0.00/ SI X/0.00/ 9° 
XS EVEN3/0 .00/,SEVEN/O .00/, ETATM/0.00/ 95/0.00/ ys SEVENI/10#0.00/,° >. © 


XSUMROW/ 100.007, SUNCLM/ 10*0.00/,A/10*0.00/,8710*0.00/5 “ 


XRATCLM/ 1006 *0/ SUBS IX/ 1000*0 00/4 MATRIX /L0000%O/ = ge 


K8y// 1/177 ,40XKe "YARLEY Wee me eg ene wand Peels Ae ence ie 


READ (5,1)TITLE 
FORMAT (2D 44) 


FORMAT (B0I1+/, 8011) 


c THIS REAOS THE DATA | 2. so, - 


00 9 I=1,NRATE . | 7 | 
READ (5,8) (NRAW(TyJ); J=1,NCASE): .  € is 


FORMAT(80I1) . a s eee 
CONT INUE ie es “ey. @ agg 
WRITEL67800) 2% & 24 e 10's 
FORMAT(*1*,55X4,"PROGRAM STANDARD" //432X,: : - ~ 


X*STATIST.IC OF CONJOINT MULTIPLE OBSERVER AGREEMENT WITH A STANDARD 
X44 933K € 
X*UNIVERSITY OF NEBRASKA-L INCOLN *y'20%» * UNIVERSITY OF PE ENNSYLVANIA', 


bps) 


“XISILTTTTTLY _ ea 
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WRITE (6,900)TITLE | - ob. es 
FORMAT ('O*,20A4) © - ‘ sf . bs 4 vi 


WRIT E( 6,90 LINCASE,NCATs VRATE mee ee: OF 
FORMAT (° O* 5X, "WITH s 4X eI 4s 2X4 "CASES" 4X9 13 92Xq. a 
JX* CAT EGORTES' 94X14 2X) OBSERVERS") ~ 

TF (NCASE.GT.160)G0 TO 500 ° ae re 3g Me. yO 
IF (NCAT.GT.10)GO'TO 503 . ” ey + _ #6 

IF (NRATE.GT.100)G0 TO 505 oe a — ae ee a ta 
WRITE(B, 902) a Ho OP ee 
FORMAT (#0"5/////,9X, "THE STANDARD CARD WAS RE EAD ast < ; 
WRITE(6,903) (KEY(S),J=1,NCASE) . e, » Hvac « 
FORMAT(* *,8011) >s 22 oe Se 
WRITES 2904). - . * 

-FORMAT (* PITT 4 Ika THE FIRST (DATA CARD WAS READ AS") > x 

I=) oe * 4 
WRIT E1105) (NRAW( Ts jisd= LeNCASS) 2, al mA 

FORMAT(’ *,BOIL) | ' .* 3 
CONT INJE << - ss | oot ee oe 
DO 20: i=1,NRATE . aA. CT oo . 
BDO 20 J=1l,NCASE . : ' : * acca 

(oy ET : 


_ XS EVENLy SEVEN2, ZET.A, SEVEN 3s SEVEN pETATMYA _—— 


READS ;4)NCASESNCATSNRATE ee 
FORMAT (13, 12913) _ os - ey Ne 
READ (5,60KEY"°  * : 3 _ = _ eo 


e 
* #% 


- : a Se Sag es pe aes >, eee ¥2 = 
, hoe : . 
’  % a ty ; : 
. - a 6 A se 
; LR(NRAW ( Ty J).£Q.0 NRAW( 5 J) =10 “ ; Be 
poe TREKEY OU) EQDDKEY(QUIFIO | eos t 
©. 20 > CONTINUE” . “  ® " 8 yo 
| NO=NGAP # NRATE © > = so. 2 . eae 
-C THIS SERTES OF CARCS(N=L THRU 60) CALCULATES. THE CELLS OF iow 
AG AGREEMENT AND DISAGREEMENT AND PLACES “THEM” IN, MATRIX oe 
me T=) . . > ~ Me et 7 . os ‘ ace 
. “M= 1 ‘ . : | ; Js : K . eae ; : og 
“my, 30 IFC IeGT»NCASE) GO TO 40 - eae ee ee Me Ue 
pO, a4 J=KEYCT) ny . © es ve ee. 2 ia 
- 4 KENRAW(N DD : re oF  & we ie 
ot VATRON.GTid) JSNCAT ® (N-L) 4 J a ne ira. 
Soe MATRIK(SPKIEMATRIX( eK) +1 = 3 @ a vot 4 ice et 
eee Or ee — > eee = = BP get 
lo GOTO .30 | oe <3 ae a, 23 ge es 
3 -40. NEN #205 2 ava ae 7 - oe. be. mes 
*. . : MaM id i + = _ 4 , * - ae - ue , > eo : . nae 
Hs? - Jel ; : . : : * * . . : . - — 
IF (Ne GT ENR ATE 1.60 10-60." . sl - me ze * 
,.'GO TO 30 “es or 2 ‘ a os fe 
+60 CONTINUE _* . = ce . - Bee 
J= hd an , ~ : 8 . as F e 
J=1 . : os , : " ae 4 : co re a 
70 TF(IeGTeNCAT) I=L | es 7, ee ee ee 
TM=TM + MATRIX( Jy) : | i te sg 
€ THIS CALCULATES THE ‘SUM OF DTAGINAL SiAGREEMENT ceLts) 2 oh ee 
I=] +1 .- ' / ot ea * * 
gage ‘oo: a ee * a5 ae as 
° TE (d.GT.NOIGO TO-73— : my 
GO TO 70 2 Fe nO 
“73 CONT INUE : 
Cc CALCULATES THE SUM DOF ROW AND TOTAL ROW VALUE - 2S 
« 00 100 I=1,NCAT . 3 . 
eS kn DO 100 JelyNCAT 
: SUMROW(T)= SUYROWCT) + WATRIX( Ted) : 
iteey: 32 ROW= ROW + MATRIX(1,J) 
Ye" 100 * CONTINUE . 
‘ * pO 120 J=ly NCAT 
7 00.120 I=1yNO . . ales 
SUMCLMES)= SUMCLM( JS) + MATRIXGT» 39 
or _ COLUMN= COLUMY + ey | . 
120 CONTINUE - : 2 . 
net OM 140 I=L NCAT 
“os ‘+ PART] = PARTI + (SUMROW CL *. SUMGLMCL)) — 
/ $40 <CONTTNUE 2% ’ % 
P ‘eTM= (1./100a) * PARTL. | 7.” _ 
“ONSeCL./(NCASEW1.)) *® PARTL | . 
; TWOrl. /JUNCASE**2 * (NCASE=1.)) 0° 6 
: _ FOUR=1./.0NCGASE -# (NCASE ~ 12D) we 
- 00 200 [=1} NCAT an. - 
“ » FIVE FIVE + SUMROWL 1)**2 * “sume LM CL) 
ae: : ; 
200 . CONTINUE = 3 -_ > 2.4 
. » * : . * 
<i « ae a " “[10}. 12 : 
¢ ENC » & : : , 
@ : ‘ . » 
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2210: TES UN. GT e NCAT} -M=M 1 3 
"Ss ate C1 Sere Nod Go TO 240° =o 


: SUBS IX1Ms J BAT CLM (Ms J) ¥#2. ca oo 


“260° CONTINUE 


180 L=1. er ate - 


' _ SIE F FOUR FIVE ~. nr ie oe 

- "DQ 240 JelgNGAT 8 
WoT NEL SCE oa tee 

: a I2l. \ ies eae 28 er te a +h e. > ane 


, Mel a . , “a Ms = : ao 7 + * 
«YF (N 2GT. NCAT) Nel ae 22g. fe hes 
-RATCEM(MyJY= RATCLA (MD) + as Ue 


‘Tet #.1> Ae rae = 


NEN EL Ane paseo” % is os 
6D TO 210 - a a TT eee 


L=1. 
fel | 7 
es re : oe. 7° 2 
. _K22 
160. IF (L eGTeNCAT) 60: 10 180, - a te tae ae Si ie 
AUJ)= CRATCLM( 12d) * RATCLAC ILD “+ RATCLM(K,3) © 6 2 
co) XRATCLMUK, LD) *® SUMROW(L) ee oe 7 re es 
. ee BLU) # ALJ) a an pe ke oe - ae: 
bab te be we Se S.C . Py 

60 TO 160 ge py %. 3 { eae 


Ao ee a rie © 2? oe 7 . — a A ee 

TF US .6T. NCAT) GD To 190 rr ee ee ee oe ge ee ee 

. =. GO TO 160 | = ' . 2 a> oe 
190 “CONTINUE ., | o , & “2 — re 
“DO 195 T=1,NCAT. | a Co Mae 
‘ZETA= “ZETA * (SUMROW(I)-* BIT). 2 : oa hee 


195 - CONTINUE. * | . 4 * See Pc. st weg? 
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260s CONT INE 


~ GO TO 599 _ 


504 _ FORMATE OS, “NUMBER OF CATEGORIES SPECIFIED ‘ON, ‘PROBLEM. - 


THREE= TwO * ZETA | ra _ -. 


>) pa 260 J= l,NEAT. > | 3 4 * 


DO 260 I=1,NRATE = ts s 
SEVENI(J)= SEVENTCI) + SUBSIXCE,J) : 


DO 270 I=1yNCAT © oe _ Se \ 
SEVEN2(L)= SEVENICI) * SUMRIWC) _ ) 


270, CONTINUE. | | - ge a. 


‘ 


DO 280 I=lyNCAT . ~ , _ a ; = 

SEVEN3= SEVEN3 +° SEVENZ( 1) me 2 | oy 

280. -CONTINUE : ok . * ee - 
_\ SEVEN= FOUR * SEVEN3 a oe in a | 

ETATM= ONE + THREE -'( FOUR * FIVE) - SEVEN & - 

G= (TM = ale a 5 = . . | 7 


500 WRITET6,501) | So & . es 
S01 FORMAT('Q*,*NJMBER OF CASES: SPECIFIED ON | PROBLEM CARD. : o 


-. *XIS TOOLARGE?). : 


- GO TO 99 | : “se 2 - = 
503. WRITE(6,504) Meee Seg . 


“2. 


5 ne ae 


y . = . : 
a A \ ¥ . 2 3 : . i 
ia . . oN * r me . 2 e , . 
= * 3 - > ee 7 . , = : a 
~ & x > ‘ “ : t 

. . Auge ‘ . ee. Ye . 8 oe a Z wes 

Fe $ a * . . x Lo > ‘ele we a Mg 7 . : eae 

7 { : i>. ¥ : - of. ‘ = x a be . * . ) Mg : . = : a é. .* . “¢ 

“XCARD 1S too “UARGE*) a O . 
tg i oy oar Ms . : 5 ‘ : oa ” 
60° Ty. O 99. 2 oe : “ a7 ; ha. a ) “a -s ’ aa . et . 


505 + WRITEt6,506) —* 

e506 * FORMATION; UNIMBER oF OBSERNERS: ‘SPECIFIED, on: pRopLem CARD 
| = XI§$ TOO LARGE) |, = ae . 
so, GO TO 99° e. : eS i ; _ 2 ; 

» 599; CONTINUE * 1.” oa Co ae 
TF" CABS (G).GE.3 5 21960 19 740... . abe e Fw tat ee 


J) CTE CABSCGNGEa 2381960 RO F207 re 
- “>. TE CABS 1G).GE62.58)G60 TO 700 yo “- 23 
~~ : 


roe TF CABSCGNGEL1096)60 TI 680 6 Fk fk my 

og Te LTR CABS(G).GE21076)60 T2660 

pA foo EF (ABS(G)+GE6E.65)G0 TD 640 ot ™. m5 

Se TER of ABS CG) LTS i. 8560 TO 600. = oa? “ee oe Se ee 

600 -WRITEC6, ao)G. | 7 Le pat ee 
B® . FORMAT('O!, "OKs" OFF 9. by tance NON-SIGNIFICANT") 8S ee 
- °, GOTO 99 °, a ee _ ae oy oe ee oe 

“640 WRITE (678456 - . * - er er es 
.84 FORMAT.('Q', 9X 'G OF '.F9. hy tenwepe P LESS-THAN 610") © 5, 0 OES 


~~, 6070 99 aoe . j re a ee a 
8660 WRITE (6.86096 5 7 we I ee eee oe. Se 
BE FORMAT('O', 9x4 'G ‘OF, Fo. Lec uaa! P LESS THAN .08") a a ee 
7 GO TO:99, cz <8 ee a ee es ee 
680 WRITE (6,88)6 oe 8 — Bae, we ee oo 
BB FORMATCIO', 9€4°G OF, A ee P LESS THAN .O5t') . = 
ons G0 TG 99 pie an 4 ee 2 a. 
“4 700 *WRITEL6,90)IG-. > . Se te ea = S ve 
8 90 FORMATO’, KG OF 4s F9- 4y tnennn= P LESS THAN OL") | - 
3 GO-TO 99 : —G _ = 
720. WRITE (6,926 ° a ' . a St ee yw 
- 92. FORMAT('O', IK ytG OFF. shy te-----.P LESS THAN .005') | ek Shy & Oy 
fe GO TO 99 2 2 2 2 :. *# Se *. 
(740° WRITE (6,9496 i. a eae . : 4 
94 FORMAT("D!, Ns *G OFS, F924, t---e-- P LESS THAN 0017) 
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